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REVEAL:  Reconstruction,  Enhancement,  Visualization, 
and  Ergonomic  Assessment  for  Laparoscopy 


1.  Introduction 

Information  cues  available  in  laparoscopy  and  other  forms  of  minimally  invasive  surgery 
are  impoverished  relative  to  cues  available  in  open  surgery.  Acquiring  surgical  skill  in 
such  an  environment  is  extremely  challenging.  Even  after  mastery,  continued  practice  can 
lead  to  problems  for  the  surgeon  as  indicated  by  frequent  incidence  of  pain  and  injury 
associated  with  laparoscopy.  The  long-term  impact  on  the  surgeon  performing  these 
procedures  is  largely  unknown. 

The  goal  of  this  work  is  to  develop  and  test  new  technologies  that  will  break  down  the 
barriers  that  block  more  surgeons  from  attaining  and  continuing  to  practice  (without 
injury  or  pain)  high  levels  of  skill  in  MIS.  This  project  will  develop  new  technology  by 
concentrating  on  three  major  research  thrusts: 

•  Smart  Image:  the  project  will  develop  and  evaluate  new  approaches  for 
extracting,  fusing,  and  presenting  information  cues  from  imagery  and  other  data 
sources. 

•  Configurable  Display:  the  project  will  develop  new  approaches  for  presenting 
existing  data  (video,  CT  data)  and  extracted  cues  (3D  reconstruction,  haptic  cues, 
etc.)  to  the  user  within  a  flexible,  configurable  display  environment 

•  Ergonomic  Assessment:  the  project  will  use  existing  technology  and  build  new 
techniques  as  needed  to  acquire  crucial  ergonomic  data  relative  to  key  factors  of 
patient  position,  technology  configuration,  and  instrument  design. 

2.  Major  Accomplishments 

In  this  section  we  provide  a  functional  view  of  major  tasks  accomplished  during  the  2004 
project  year.  These  include  (1)  creation  of  the  software  development  laboratory,  (2) 
distribution  of  REVEAL  Suite  release  1.0,  (3)  distribution  of  REVEAL  Suite  release  2.0, 
(4)  large-scale  projection  of  stereo  images  from  a  laparoscopic  probe,  (5)  creation  of  a 
UMM  research  operating  room  suite,  (6)  securing  Human  Studies  IRB  approval,  and  (7) 
creation  of  a  study  framework  for  quantitative  analysis  of  latency  on  task  performance. 

Creation  of  Software  Development  Laboratory 

The  software  development  laboratory  was  established  at  the  beginning  of  the  year.  It 
consists  of  a  distributed  rendering  system  containing  a  22  node  cluster  computer  and  a 
workstation  for  use  as  a  coordinating  “head  node.”  Two  additional  workstations  were 
acquired  for  use  by  the  development  staff.  The  workstations  run  RedHat  Enterprise 
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Linux  OS.  We  have  also  purchased  12  DLP  projectors  to  create  the  large  format  display. 
Recently  we  purchased  a  rear  projection  polarity  preserving  screen  for  stereo  display. 

The  screen  and  projectors  are  mounted  on  a  custom  made  truss.  Low  cost  video  cameras 
are  used  to  calibrate  the  projectors. 

A  dual  Xeon  processor  server  is  also  part  of  the  laboratory.  The  server  is  equipped  with  a 
tape  drive  and  a  dual-disk  RAID  controller.  The  server  runs  RedHat  Advanced  Server 
OS.  We  run  a  number  of  services  on  the  server  including  (1)  the  website  for  the 
REVEAL  project;  (2)  the  CVS  server  and  repository;  (3)  a  MySQL  database  server  and 
repository;  (4)  an  NFS  networked  file  system  server;  (5)  and  an  LDAP  user 
authentication  server;  and  (6)  centralized  tape  backup  of  all  critical  data.  The  server  also 
performs  regular  nightly  builds  and  tests  on  the  most  current  version  of  the  application  in 
CVS. 

Laboratory  machines  are  interconnected  using  a  gigabit  network  switch,  providing 
reduced  latency  and  increased  throughput  relative  to  current  10/100  Ethernet  switches. 

All  laboratory  machines  are  on  a  dedicated  network  connected  to  the  larger  university 
network  through  a  hardware  firewall.  Protection  of  laboratory  resources  by  a  firewall 
reduces  the  potential  for  infiltration  by  hackers  or  denial  of  service  attacks. 

A  number  of  software  development  policies  and  procedures  were  drafted  and 
implemented  at  the  inception  of  the  laboratory.  These  include  (1)  work  practices  and 
coding  standards;  (2)  backup  procedures;  (3)  nightly  build  and  regression  test  procedures; 
and  (3)  protocols  for  using  a  common  software  repository  (CVS). 

REVEAL  Suite  Release  1.0 

Release  1.0  of  the  REVEAL  tools  provides  large-scale  high-resolution,  high-brightness 
display  of  OpenGL  applications  and  video  using  an  arbitrary  number  of  low-cost  LCD  or 
DLP  projectors.  It  is  based  on  the  VIBE  tool,  previously  developed  by  researchers  at  the 
University  of  Kentucky,  with  ancillary  applications  created  to  integrate  the  general 
purpose  VIBE  tool  into  a  context  appropriate  for  clinical  use. 

Using  VIBE,  an  arbitrary  number  of  projectors  can  be  tiled  together  to  create  a  single 
large  scale  display.  The  calibration  algorithm  used  is  robust,  efficient  and  flexible.  The 
projectors  can  be  positioned  casually  and  need  not  be  aligned  in  any  way.  The  projection 
surface  can  have  any  arbitrary  shape  and  is  not  required  to  be  flat.  By  using  binary 
encoded  patterns  the  calibration  time  is  reduced  to  logarithmic  order  per  projector.  As 
part  of  our  development  activities,  VIBE  has  been  enhanced  with  background  subtraction 
technique  to  reduce  errors  during  the  calibration  process.  This  also  lets  us  project  on 
surfaces  with  imperfections. 

The  release  also  includes  the  SmartVideo  application,  providing  real  time  video  at  ~30 
frames  per  second  on  a  gigabit  network.  Written  using  OpenGL,  the  SmartVideo 
program  can  play  real-time  video  on  the  tiled  display  using  the  information  gathered  by 
the  calibration  program  to  produce  a  single,  uniform  frame.  Although  Chromium  can  run 
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most  off-the-shelf  OpenGL  video  applications,  the  frame  rate  obtained  by  using  them  on 
general-purpose  hardware  is  very  low.  The  SmartVideo  application  has  been  optimized 
to  run  efficiently  on  the  cluster  and  network  driving  the  tiled  display  and  offer  higher 
frame  rates.  This  is  achieved  by  fragmenting  the  captured  video  frames  and  sending  only 
the  required  fragments  to  the  appropriate  projector.  The  number  and  size  of  fragments  is 
computed  using  the  calibration  information.  SmartVideo  is  also  designed  to  preserve  the 
aspect  ratio  on  the  tiled  display  based  on  dynamically  detected  calibration  parameters. 

A  user  friendly,  portable,  Java  GUI  is  also  provided  with  the  suite.  The  GUI  can  be  used 
to  configure  the  calibration  parameters  and  launch  OpenGL  applications  without  the  need 
to  manually  update  low-level  configuration  files. 

REVEAL  Suite  Release  2.0 

Release  2.0  of  the  REVEAL  tools,  released  in  the  middle  of  October  2004,  provides  a 
number  of  enhancements  over  version  1.0.  The  robustness  and  reliability  of  the 
calibration  process  has  been  improved.  It  now  also  includes  a  feature  to  perform  a  single- 
step  fast  calibration  that  works  for  all  kinds  of  surfaces  but  with  limited  projector 
orientation.  It  assumes  that  the  projectors  are  oriented  such  that  their  center-line  of 
projection  is  orthogonal  to  the  projection  surface.  This  reduces  calibration  time  to  a 
constant  value  per  projector.  The  calibration  process  has  also  been  enhanced  to  adapt  to 
situations  where  camera  and  viewer  are  positioned  on  opposite  sides  of  the  projection 
surface  at  the  time  of  calibration. 

The  calibration  software  in  this  release  can  perform  stereo  calibration,  calibrating  two 
independent  pools  of  tiled  projectors  so  they  can  display  stereo  images.  If  the  projectors 
are  oriented  suitably,  fast  stereo  calibration  can  also  be  done.  The  calibration  tries  to 
maximize  the  overlap  area  available  for  stereo  display. 

In  addition  to  the  SmartVideo  application,  a  SmartStereo  application  is  provided  with  this 
release.  The  SmartStereo  program  can  play  real  time  stereo  video  on  the  stereo  calibrated 
projectors.  Synchronized  video  grabbing  is  included  in  the  SmartStereo  application. 
Synchronized  capture  ensures  that  left  and  right  images  are  captured  at  as  close  to  the 
same  instant  as  possible.  SmartStereo  can  play  synchronized  stereo  video  at  12-14  fps 
using  the  cluster  computer  and  network  configuration  described  above. 

Large-Scale  Projection  of  Stereo  Images  from  Laparoscopic  Probe 

Large  scale  projection  of  stereo  images  is  a  seamless  stereo  display  formed  by  using  two 
pools  of  calibrated  projectors.  The  display  is  used  to  render  high  resolution  stereo  images 
with  depth  cues,  thus  creating  a  realistic  3D  view  for  the  user.  The  display  can  be  used  in 
multi-viewer  environment  and  also  for  training  purposes. 

A  large-scale  projection  system  of  stereo  images  is  a  complex  system  involving  many 
hardware  and  software  components.  The  hardware  components  of  the  system  include  two 
or  more  projectors,  a  camera,  a  gigabit  network,  a  cluster  of  computers,  each  of  which  is 
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connected  to  a  projector,  a  projection  screen,  polarized  viewing  goggles,  polarized  filters, 
and  heat  resistant  custom  designed  filter  mounts.  The  software  components  include 
proprietary  stereo  calibration  software,  the  Chromium  distributed  rendering  software 
framework,  and  the  custom  built  OpenGL  stereo  applications. 


Scope  Computer 


Cluster  of  Projectors  Rear  Projected 

Computers  Screen 


Pipeline  of  the  large  scale  stereo  image  projection  system 


The  projectors  are  divided  into  two  sets,  each  set  representing  either  a  left-eye  view  or  a 
right-eye  view.  Each  set  of  projectors  is  independently  calibrated  using  the  functionality 
of  VIBE  calibration  described  above.  The  calibration  information  of  the  two  views  is 
processed  and  the  stereo  parameters  are  calculated.  The  OpenGL  stereo  applications  use 
these  stereo  parameters  to  render  the  stereo  display.  The  user  only  has  to  designate  each 
projector  as  either  left  or  right.  No  further  user  interaction  is  needed  to  calibrate  and 
render  the  stereo  display.  Synchronization  is  provided  at  the  image  capture  end  and  as 
well  as  at  the  image  rendering  end  of  the  pipeline. 

With  eight  projectors  and  eight  render  nodes,  a  4X4  tiled  stereo  display  is  formed  with  a 
resolution  of  about  2000  X  1500  pixels.  In  our  current  system  implementation  the  frame 
rate  on  gigabit  network  is  around  12-14  frames  per  second.  We  are  in  the  process  of 
analyzing  the  performance  and  sensitivity  of  the  current  system  with  a  goal  of  producing 
stereo  video  in  the  25-30  frames  per  second  range. 

Creation  of  UMM  Research  Operating  Room  Suite 

We  have  secured  space  in  the  University  of  Maryland  Medical  Center  for  deployment  of 
the  experimental  environment  to  be  used  for  simulation,  testing,  and  the  human  study 
protocol.  This  space  is  located  in  the  old  Operating  Room  suite  and  has  the  advantage  of 
being  equipped  as  a  functional  OR  in  terms  of  floor  plan  and  dimension.  This  suite  will 
be  used  as  the  major  deployment  platform  in  2005  for  the  technology  we  have  developed 
to  be  used  in  the  human/animal  protocol  and  assessment  study  to  follow. 

Human  Studies  IRB  Approval 
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We  failed  in  2004  to  complete  the  IRB  approval  process  through  TATRC  and  the  US 
Army.  Until  this  approval  is  received  we  are  unable  to  run  the  protocol  as  it  relates  to 
human  subjects  and  animals.  We  nevertheless  have  moved  forward  with  technical 
development,  space  and  infrastructure  acquisition,  and  readiness  and  planning  to  carry 
out  the  protocol  once  approvals  are  in  place. 

Quantitative  Analysis  of  Latency  on  Task  Performance 

We  have  begun  a  secondary  task  analysis  and  a  latency  task  performance  study  to  more 
closely  define  the  parameters  that  will  likely  affect  performance  of  the  new  technology 
(especially  display  settings).  This  analysis  will  inform  system  settings  and  help  us 
establish  the  guidelines  for  the  human  study  that  will  bring  out  the  most  effective  results. 


3.  Project  Milestones  2004 

The  2004  project  plan  included  two  distinct  sets  of  milestones,  one  for  visualization 

technology  development  activities,  and  a  second  for  ergonomic  assessment  activities. 

The  milestones  and  our  progress  in  reaching  each  are  assessed  in  the  sections  that  follow. 

Primary  Milestones:  Visualization  Technology 

1)  Hire  and  train  development  personnel 

In  November  of  2003  two  “Program  Systems  Analyst”  positions  were  posted 
through  the  University  of  Kentucky  hiring  system.  A  total  of  35  applications 
were  received  and  vetted.  Of  these  35  applicants  three  were  found  to  fully  satisfy 
all  qualifications  outlined  in  the  original  job  posting.  These  three  were  invited  for 
interviews.  Based  on  these  interviews  two  candidates  were  selected  to  receive  the 
first  offers.  Both  accepted,  successfully  negotiated  contracts,  and  started  work  on 
the  project. 

Yogesh  Shukla,  MS,  joined  the  project  as  “Program  Systems  Analyst”  on  15 
December  2003.  Praveen  Devabhaktuni,  MS,  joined  the  project  as  “Program 
Systems  Analyst”  on  5  January  2004.  Both  have  been  actively  involved  in  (1) 
establishing  the  project’s  laboratory  and  software  development  environment;  and 
(2)  developing  significant  new  functionality  to  support  large-scale  projection  of 
monocular  and  stereoscopic  images  from  laparoscopic  camera  probes. 

In  addition  to  the  full-time  staff  we  have  hired  an  undergraduate  student,  Ryan 
Davis,  in  a  supporting  role.  Initially  he  was  responsible  for  maintaining  the 
project  web  site  and  cataloging  project  media  (still  photos,  videos,  etc.).  His  work 
in  this  area  was  performed  with  such  efficiency  and  high  quality  that  lately  we 
have  been  transitioning  him  into  a  role  that  will  involve  him  in  an  undergraduate 
research  experience.  Currently  he  is  involved  in  defining  and  developing  tools  to 
support  set-up  and  calibration  of  multi-projector  displays. 
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During  the  summer  of  2004  we  employed  a  graduate  student,  Michael  Jacobs 
from  New  York  University,  who  focused  on  two  major  tasks:  (1)  developing  tools 
to  support  experiments  to  determine  the  impact  of  latency  on  performance  of 
minimally  invasive  surgical  tasks;  and  (2)  researching  techniques  for  controlling 
program  functionality  that  could  be  integrated  into  laparoscopic  surgical 
procedures  with  little  or  no  additional  equipment  and  no  negative  impact  on 
patient  safety  or  clinician  performance. 

2)  Establish  software  development  environment  (hardware,  software,  standards) 

Early  in  2004  we  purchased  computing  equipment  (a  22-processor  cluster 
computer,  desktop  computers,  networking  infrastructure,  etc.)  to  support  research 
and  development  activities.  Three  workstations  were  purchased,  two  for  the 
development  staff  and  another  to  act  as  the  head  node  to  control  the  cluster 
computer. 

The  development  staff  configured  and  deployed  this  equipment  to  create  (1)  a  test 
environment  for  laparoscopic  capture  and  multi-projector  display  R&D,  and  (2)  a 
development  environment  to  support  software  development  activities.  A 
description  of  the  software  development  environment  follows  in  Section  4. 

The  development  staff  worked  together  to  evolve  a  set  of  standards  to  be  followed 
while  working  on  the  project.  Specifically  the  Scrum  methodology  for  agile 
software  development  has  been  adopted.  This  methodology  is  suitable  for 
research  project  like  ours  where  new  research  ideas  and  hypothesis  need  to  be 
tested  quickly  to  determine  their  feasibility  before  working  to  integrate  them  in 
the  final  product.  The  entire  team  meets  every  month  to  select  items  from  the 
project  backlog  that  can  be  implemented  within  a  thirty  day  iteration  called  a 
“sprint.”  The  idea  is  to  focus  on  immediate  goals  and  let  the  architecture  and 
design  evolve  over  multiple  sprints.  In  addition  the  development  team  holds  daily 
meetings  to  analyze  progress  and  hurdles.  Improvements  and  alternatives  are 
discussed  during  these  meetings.  Team  members  regularly  document  problems  as 
well  as  successes  in  a  bug  tracking  database.  Coding  standards  and  testing 
procedures  have  been  developed  which  guide  the  software  development  process. 

3)  Establish  MIS  test-bed  (collect  instruments,  interfaces) 

We  have  created  an  MIS  test-bed  on  which  new  ideas  are  prototyped  and 
integrated  into  an  evolving  suite  of  tools.  Instruments  and  devices  were  acquired 
from  various  sources  to  build  the  test-bed.  A  laparoscopic  trainer  stand  was 
obtained  from  Stryker.  It  is  used  as  a  crude  simulation  of  a  laparoscopic  surgical 
site,  to  help  medical  students  become  accustomed  to  the  limited  visual  cues 
available  in  MIS  procedures,  and  develop  manual  dexterity  in  the  use  of  MIS 
instruments.  In  addition,  Stryker  has  also  provided  various  surgical  tools  (clamps, 
probes,  etc.)  used  for  laparoscopy.  Camera  probes  specialized  for  this  kind  of 
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surgical  operation  as  well  as  controllers  and  light  sources  are  also  part  of  the  test¬ 
bed. 

Recently  we  acquired  a  stereoscopic  camera  probe  from  Viking  Systems 
(formerly  Vista  Surgical  Systems)  to  facilitate  the  development  of  stereo  vision 
applications.  A  mannequin  has  also  been  acquired  as  part  of  our  effort  to  develop 
a  system  for  human  anatomical  studies  based  on  non-planar  projective  displays. 

The  overall  goal  of  developing  the  test-bed  is  to  create  a  simulation  environment 
where  new  display  and  image  processing  technologies  can  be  tested  and  evaluated 
during  the  development  process. 

4)  Gather  product  requirements 

In  keeping  with  the  Scrum  methodology,  we  have  approached  requirements 
gathering  as  an  ongoing  process,  rather  than  single  monolithic  phase.  Throughout 
the  year  formal  and  informal  meetings  between  software  developers  at  the  UK  site 
and  personnel  focused  on  ergonomics  issues  at  UMM  have  been  held.  During 
those  meetings  the  needs  of  UMM  researchers  have  been  elicited  and 
reformulated  as  potential  new  features  for  the  REVEAL  suite  of  tools  being 
developed  by  the  UK  researcher  and  development  team. 

5)  Analyze  product  requirements 

Requirements  gathered  as  described  in  (4)  above  have  been  reformulated  as 
individual  implementable  features  and  entered  into  the  project  backlog,  a 
repository  for  maintaining  a  list  of  known  bugs  and  potential  features.  Each 
month  this  backlog  is  assessed,  the  items  are  prioritized,  and  items  are  selected  for 
implementation  as  part  of  the  month’s  development  activities. 

6)  Hold  a  multi-site  project  workshop  meeting 

On  November  1 7th  UK  and  UMM  researchers  associated  with  the  REVEAL 
project  met  for  a  multi-site  project  workshop  at  the  University  of  Maryland 
Medical  Center.  During  this  meeting  progress  toward  the  year’s  milestones  was 
assessed,  presentations  were  made  covering  significant  results  achieved,  and  plans 
were  made  for  continuing  progress  into  the  next  year  and  continuing  inter-site 
coordination. 

7)  Define  a  scalable  real-time  architecture  for  multi-input,  multi-transformation, 
multi-display  MIS  support 

The  diagram  below  describes  the  architecture  of  the  display  system  that  has  been 
developed.  We  create  a  large  format  display  using  an  array  of  arbitrarily 
positioned  projectors  which  are  calibrated  using  the  VIBE  software.  Each 
projector  is  driven  by  a  node  in  the  cluster.  A  head  node  works  as  the 
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coordinator,  determining  how  to  split  and  distribute  the  display  stream  among  the 
projectors. 


The  head  node  and  the  cluster  nodes  are  connected  through  a  gigabit  network  to 
ensure  minimum  latency  in  the  distribution  of  the  display  stream.  We  run 
Chromium  on  the  head  node  and  also  on  each  of  the  cluster  nodes.  The  OpenGL 
compatibility  of  Chromium  ensures  that  most  applications  can  be  run  on  this 
architecture  without  any  changes.  On  the  head  node  Chromium  uses  the 
calibration  information  to  split  and  distribute  the  display  stream.  VIBE  adds  an 
additional  stream  processing  unit  (SPU)  to  Chromium  at  the  cluster  end  which 
receives  the  display  stream  from  the  head  node  and  applies  warping  and  blending 
using  the  calibration  information  to  create  a  large  tiled,  seamless  display. 

8)  Identify  re-usable  components  from  public  domain  and  outside  researchers 

Identifying  externally  developed,  reusable  components  has  had  several 
advantages.  First,  the  use  of  components  that  perform  generic  tasks  has  allowed 
us  to  concentrate  on  tasks  specific  to  our  domain.  Second,  use  of  industry 
standard  tools  opens  a  wide  range  of  input  sources  to  us  without  any  effort  on  our 
part. 

The  principal  public  domain  components  that  we  have  adopted  are  as  follows: 

OpenGL  -  An  industry  standard  graphics  library  and  API.  It  is  used  by 
application  programs  as  a  common  interface  for  generating  elaborate  graphics 
displays  from  3D  models.  Using  OpenGL  as  the  graphics  library/API  for  our 
project  has  allowed  us  to  incorporate  a  number  of  existing  general-purpose  and 
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medical  imaging  software  applications  into  our  system  with  no  programming 
effort. 

Chromium  -  An  OpenGL  compatible,  generic,  public  domain  software  system  for 
distributed  rendering.  Chromium  is  the  framework  that  supports  our  projective 
display  system — a  single  video  source  is  received,  divided  by  a  small  application 
program,  processed  through  the  Chromium  distributed  rendering  system,  and 
displayed  using  another  small  application  program  from  the  VIBE  system. 

OpenCV  -  A  public  domain  library  containing  implementations  of  standard 
computer  vision  algorithms.  Use  of  OpenCV  has  enabled  our  development  team 
to  concentrate  on  novel  applications  of  computer  vision  techniques  to  our  problem 
domain,  leveraging  tried  and  tested  implementations  of  classic  algorithms  with 
minimal  effort. 

CVS  -  Use  of  CVS  controls  concurrent  access  to  source  code  and  documentation 
files,  and  creates  a  repository  that  organizes  a  multi-release  project  into  discrete 
“versions.”  This  enables  simultaneous  development  by  the  staff  and  versioning  of 
ongoing  work. 

Mantis  -  To  document  and  maintain  the  project  backlog  required  by  Scrum  we 
use  the  Mantis  bug  tracking  system.  The  development  staff  track  tasks  assigned  to 
them  and  record  progress  and  problems  regularly  using  Mantis. 


9)  Implement  first  iteration  of  architectural  framework 

The  goal  of  the  first  iteration  was  to  create  a  flexible,  low  cost,  large  format 
display  system.  We  have  used  inexpensive  off-the-shelf  projectors  and  hardware. 
Another  objective  was  to  create  a  generic  display  system  that  would  support  a 
wide  array  of  applications.  Keeping  the  system  flexible  meant  that  the  projectors 
were  not  required  to  be  aligned  in  any  rigid  way.  They  could  be  tiled  arbitrarily 
and  calibrated  to  create  a  large  seamless  display. 

As  part  of  the  first  year’s  effort  we  implemented  an  efficient  and  robust  projector 
calibration  technique  for  monocular  and  stereo  projection.  We  also  developed  a 
real  time  video  application  for  the  tiled  display.  A  synchronized  stereo  video 
application  has  been  developed  that  can  run  on  the  stereo  calibrated  system. 
Further  details  of  our  system  are  presented  below. 

10)  Implement  flexible  display  back-end  for  unprocessed  probe  camera  data 

We  have  implemented  our  first  iteration  of  a  display  back-end.  This  work 
encompasses  the  SmartVideo  and  SmartStereo  applications.  These  tools  provide  a 
framework  for  (1)  efficient  display  of  unprocessed  probe  camera  data,  and  (2) 


12 


future  integration  of  computer  vision  techniques  to  improve  and  enhance  probe 
camera  images  in  real-time  and  near-real  time. 

Primary  Milestones:  Ergonomic  Assessment 

1)  Obtain  UMD  IRMACUC  approval 

2)  Hire  and  train  development  personnel 

3)  Establish  ergonomic  assessment  environment  (space,  hardware,  software, 
assessment  methodologies) 

4)  Organize  human  subjects  for  baseline  tasks 

5)  Analyze  assessment  hardware/software  system  requirements 

6)  Perform  baseline  assessment  study  using  human  subjects  completing  tasks  in 
training  environment 

7)  Define  technical  requirements  for  improved  assessment  technology 

8)  Process  and  interpret  baseline  study  using  trainer  boxes  and  assessment 
environment 

9)  Hold  a  multi-site  project  workshop  meeting 

10)  Conduct  follow-on  study  with  humans  completing  laparoscopic  baseline  tasks  on 
animals 

4.  Project  Milestones  2005 

The  2005  project  plan  again  included  two  distinct  sets  of  milestones  for  visualization 
technology  development  activities  and  ergonomic  assessment  activities.  The  milestones 
and  our  plans  for  reaching  each  during  the  coming  year  are  assessed  in  the  sections  that 
follow. 

Primary  Milestones:  Visualization  Technology 

1)  Deploy  and  evaluate  first  iteration  of  cluster-based  distributed  architectural 
framework 

The  November  multi-site  project  workshop  included  planning  for  the  deployment 
of  UK  REVEAL  developed  processing  and  display  systems  in  the  UMM  research 
operating  room  suite.  Deployment  is  planned  for  early  2005. 

2)  Deploy  and  evaluate  display  back-end  for  probe  camera  data 

The  November  multi-site  project  workshop  included  planning  for  the  deployment 
of  UK  REVEAL  developed  processing  and  display  systems  in  the  UMM  research 
operating  room  suite.  Deployment  is  planned  for  early  2005. 

3)  Integrate  stereo  probe  device  support  into  acquisition  and  back-end  display 
framework 
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Work  on  this  task  commenced  in  late  2004  and  we  have  already  nearly  completed 
a  first  prototype  of  a  straight-through  stereo  video  display  system  for  the  Viking 
Systems  probe  and  controller. 

4)  Design  and  test  algorithms  for  low-latency  probe-data  cue  extraction 
(reconstruction,  enhancement,  overlays) 

Some  preliminary  work  on  user-interface  design  issues  was  carried  out  by  the 
summer  student  in  2004.  Work  is  scheduled  to  continue  in  2005. 

5)  Design  and  test  algorithms  for  non-invasive  ergonomic  cue  extraction  in 
simulation/surgical  setting 

Work  is  scheduled  to  commence  in  2005  with  additional  input  from  UMM 
researchers  and  clinicians. 

6)  Design  display  mode  alternatives  for  same-display  integration  of  probe-data, 
extracted  cue  data  and  other  overlay  information 

Work  is  scheduled  to  commence  in  2005. 

7)  Test  and  support  performance  analysis  of  trial  environment  configurations:  stereo, 
display  configurations,  integrated  cues 

Performance  analysis  and  management  activities  commenced  in  2004  and  are 
ongoing.  We  are  actively  analyzing  the  performance  of  various  configurations, 
and  creating  requirements  for  enhancements  required  for  more  accurate  and 
timely  reporting  of  performance  statistics  in  future  releases  of  the  REVEAL  suite. 

Primary  Milestones:  Ergonomic  Assessment 

1)  Organize  performance  trials  on  baseline  tasks 

2)  Assess  latency  impact  of  distributed  architecture 

3)  Upgrade  hardware/software  environments 

4)  Conduct  implementation  tests  on  stereo-probe  acquisition  and  display 

5)  Design  and  perform  stereo-based  human  skills  study 

6)  Design  and  test  non-invasive  assessment  algorithms 

7)  Design  and  perform  human  skills  study  with  trial  environment  configurations: 
display,  integrated  cues,  stereo 

5.  Summary 

We  met  all  technical  milestones  in  2004  and  are  eager  to  move  forward  with  the  plan  for 
2005.  We  have  been  hampered  on  the  clinical/human  subjects  side  of  the  project  because 
we  have  still  not  obtained  final  approval  for  our  protocols.  We  envision  accelerated 
forward  progress  once  these  approvals  are  in  place,  and  we  anticipate  that  to  occur  early 
in  Project  Year  3  (early  2005). 
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Appendix  A:  Project  Personnel 


Name 

Role 

Location 

2005 

FTE 

W.  Brent  Seales,  PhD 

Principal  Investigator 

UK  College  of  Engineering 

31.5% 

Adrian  Park,  MD 

Co-Principal 

Investigator 

UM  School  of  Medicine 

Tsegay  Baraki 

Administrative  Support 
(Budget,  Reporting) 

UMMS  Department  of 
General  Surgery 

Jesus  Caban 

Research  Assistant 

UK  Department  of  Computer 
Science 

100% 

C.  Melody  Carswell 

Senior  Researcher 

25% 

Duncan  Clarke,  PhD 

Technical  Project  Lead 

Fremont  Associates,  LLC 

60% 

Ryan  Davis 

Student  Programmer 

UK  Center  for  Visualization 

Praveen  Devabhaktuni, 
MS 

Program  Systems 

Analyst 

UK  Center  for  Visualization 

100% 

Ivan  George 

Technical  Support 

Hi 

Kimberly  Hall 

Administrative  Support 
(Budget,  Clerical) 

UK  Center  for  Visualization 

20% 

Stephen  Kavic,  MD 

Senior  Researcher 

UM  School  of  Medicine 

Huey  Khoo 

Research  Assistant 

UK  Department  of  Computer 
Science 

100% 

Linda  Rice,  RN,  CCRC 

Administrative  Support 
(Research  Protocols) 

UK  Medical  Center 

20% 

Ross  Segan,  MD 

Senior  Researcher 

UM  School  of  Medicine 

Robert  Shapiro,  PhD 

Senior  Researcher 

UK  Department  of 
Kinesiology  and  Health 

5% 

Yogesh  Shukla,  MS 

Program  Systems 

Analyst 

UK  Center  for  Visualization 

100% 

Donald  Witzke,  PhD 

Senior  Researcher 

UK  Department  of  Pathology 

5% 

Appendix  B:  Laboratory  Facilities 

UK  Software  Development  Laboratory 

•  Project  Staff  Office 

o  Location:  801  KU  Building 

o  Purpose:  Working  environment  for  day-to-day  activities  of  software 
developers, 
o  Equipment: 

■  Two  developer  workstations 

■  Twin-processor  Dell  PowerEdge  server  with  DLT  tape  drive 

■  Cisco  firewall/router 

•  High-Performance  Multi-Projector  Display  Laboratory 

o  Location:  871  KU  Building 

o  Purpose:  Test  environment  for  MIS  video  image  processing  techniques 
and  large-scale  projected  displays, 
o  Equipment: 

■  RackSaver  22  Processor  cluster  computer  with  gigabit  backplane 

■  Dell  workstation 

■  Gigabit  network  switch 

■  12  DLP  projectors  with  overhead  mounts 

■  Custom  heat-  and  vibration-tolerant  filters  w/  mounts  for  stereo 
projection 

■  7.5’  x  10’  back-projected  polarity-preserving  screen  with  mounting 
frame 

■  General  purpose  Canon  video  camera 

■  Stryker  trainer  stand  with  auxiliary  high-definition  LCD  display 

■  2  Stryker  888  high  resolution  cameras,  controllers  and  lens  probes 

■  Stryker  light  source 

■  Viking  Systems  stereo  camera  probe,  controller  and  light  source 

■  Assorted  MIS  surgical  instruments 


•  Project  Office 

o  Location:  883  KU  Building 

o  Purpose:  Working  environment  for  project  management  and  small  team 
meetings. 

o  Equipment:  One  general  purpose  computer. 

•  Web  site 

o  Location:  http://halsted.vis.uky.edu 

o  Purpose:  Provide  general  overview  of  project  activities,  distribute  project 
documents  and  software,  and  serve  as  repository  for  project  images  (still 
and  video). 


UMM  Research  Operating  Room  Suite 


•  Simulation  space  in  old  OR  Suite 

o  Location:  UMB  Medical  Center 

o  Purpose:  Working  environment  for  day-to-day  activities  of  human  subjects 
using  technology  and  simulation  environment 

■  Renovations  for  power,  computer  cluster  mounting,  network 
backbone 

■  Mounting  hardware  for  projection,  screen  surface  material, 
cameras  for  observation 


Appendix  C:  Publications 

J  Caban,  W.B.  Seales,  A.  Park.  Heterogeneous  Displays  for  Surgery  and  Surgical 
Simulation,  MMVR  2005,  Long  Beach,  CA. 

A.  Park.  A  novel,  comprehensive  model  of  paraesophageal  hernias,  SAGES  2005. 

A.  Park.  3-Dimensional  Modeling  based  on  Computed  Tomography  (CT)  Data  for 
Evaluation  and  Preoperative  Planning  in  Adrenalectomy:  Surgically-Relevant 
Imaging,  SAGES  2005. 

A.  Park.  A  novel  conceptual  model  of  the  current  surgical  classification  of 
paraesophageal  hernias  using  dynamic  three-dimensional  reconstruction.  SAGES 
2005. 

A.  Park.  Repair  of  a  complex  foregut  hernia  aided  by  novel  three-dimensional 
surgical  reconstruction.  SAGES  2005 

A.  Park.  The  application  of  three-dimensional  imaging  in  planning  operative 
treatment  of  gastrointestinal  stromal  tumors.  (SAGES  2005,  submitted  in 
video) 


